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Introduction 

Wide-area  motion  imaging  (WAMI)  has  received  increased  attention  in  defense  and  commercial 
space  due  to  the  importance  of  wide-area  persistent  surveillance  for  homeland  protection, 
battlefield  situational  awareness,  ISR  of  denied  areas,  and  environmental  monitoring.  Recently 
developed  systems  such  as  Argus-IS  have  the  capability  to  surveil  up  to  100km2  at  over  a 
gigapixel  resolution  from  an  airborne  platform.  This  huge  amount  of  visual  data  requires 
algorithms  for  automated  detection  and  tracking  of  targets  of  interest.  However,  traditional 
kinematic  data  based  tracking  algorithms  have  challenges  in  wide  area  motion  imagery  due  to  a 
relatively  low-sampling  rate,  low  spatial  resolution,  occlusions,  changes  in  lighting,  and  multiple 
confusers.  Recent  studies  have  shown  that  incorporating  hyperspectral  data  can  boost  probability 
of  detection,  reduce  false  alarms  and  improve  performance  in  vehicle  tracking  and  dismount 
detection  [1], 

Currently  fielded  imaging  spectrometers  use  either  dispersive  or  interferometric  techniques.  A 
dispersive  spectrometer  uses  a  grating  or  prism  to  disperse  the  spectrum  along  one  axis  of  a  focal 
plane  array  (FPA)  while  the  other  axis  is  used  to  measure  a  single  spatial  dimension.  An 
interferometric  spectrometer  reconstructs  the  spectrum  from  multiple  interferograms  measured  at 
the  FPA  by  splitting  the  incident  light  into  two  optical  paths  and  varying  the  optical  path  distance 
of  one  of  the  paths  with  a  moveable  mirror.  Both  approaches  are  not  suitable  for  motion  imaging 
a  large  area  on  the  ground.  For  example,  to  cover  64  km2  at  a  ground  sampling  distance  of  0.5m, 
an  update  rate  of  1Hz,  and  up  to  256  spectral  bands,  a  dispersive  grating  spectrometer  must 
sacrifice  SNR  (<4us  dwell  time  per  pixel).  An  interferometric  spectrometer  is  not  even  capable 
of  imaging  at  a  1  Hz  update  rate  as  it  would  require  it’s  mirror  to  move  an  order  of  magnitude 
faster  (65000  steps/sec)  than  what  is  typically  available  (2000  steps/sec)  .  Given  these 
constraints,  it  is  not  surprising  that  no  military  or  commercial  WAMI  platform  has  a 
hyperspectral  sensing  capability.  Therefore,  today’s  systems  must  choose  between  area  coverage 
and  spectral  bandwidth.  Time-encoded  multiplexed  imaging  has  the  potential  to  enable  wide  area 
hyperspectral  motion  imaging  as  it  has  greater  throughput  than  a  dispersive  imager  and  a  faster 
scan  rate  than  an  interferometric  imager. 

Time-encoded  Multiplexed  Imaging 

The  key  idea  behind  time-encoded  multiplexing  is  mapping  spectral  features  in  the  scene  to 
orthogonal  temporal  codes.  To  illustrate  this  concept,  consider  a  single  spatial  pixel  with  the 
understanding  that  each  pixel  operates  independently,  thus  this  technique  can  scale  to  any  size 
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array  of  pixels.  In  Figure  1  (top),  a  single  spatial  pixel  contains  three  spectral  colors:  red,  green, 
and  blue.  These  colors  will  be  assigned  the  orthogonal  codes  {0,1,1},  {1,0,1},  and  {1,1,0}.  The 
light  is  dispersed  through  a  first  prism  onto  three  pixels  on  a  spatial  light  modulator  then 
recombined  and  measured  at  a  single  pixel  detector.  During  the  integration  period,  the  three  codes 
are  sequenced  and  three  measurements  taken.  During  the  first  time  sequence  (ti),  the  spatial  light 
modulator  is  set  to  the  first  code  {0,1,1}  which  blocks  the  red  light  so  the  measurement  mi  is  a 
sum  of  green  and  blue.  This  is  repeated  for  the  subsequent  codes  for  a  total  of  three 
measurements.  An  estimate  of  the  amount  of  red,  green,  and  blue  light  within  the  pixel  can  be 
calculated  by  addition  or  subtraction  of  the  measurements.  For  example,  the  blue  channel  is  the 
addition  of  the  first  two  measurements  and  subtraction  of  the  third  measurement. 
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Figure  1  Image  encoding  with  a  spatial  light  modulator  (top)  and  decoding  with  a  digital  focal  plane 
array  (bottom). 


The  image  decoding  can  be  performed  independent  from  the  measurement  by  reading  out  an 
image  frame  for  each  time  sequence,  however,  the  frame  rate  of  the  imager  will  limit  the  image 
decoding  rate,  which  will  limit  the  hyperspectral  data  (hypercube)  acquisition  rate.  For  example, 
at  100  frames/sec  and  200  spectral  channels,  the  acquisition  rate  is  0.5Flz. 
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Implementing  decoding  with  a  digital  focal  plane  array  (DFPA)  enables  much  faster  hypercube 
acquisition  rate  because  the  decoding  can  be  performed  in  parallel  and  at  the  same  time  as  the 
measurement.  In  a  digital  focal  plane  array  (bottom  of  Figure  1),  each  pixel  has  an  analog-to- 
digital  converter  (ADC).  The  ADC  converts  the  input  photocurrent  into  a  digital  pulse  stream, 
and  a  counter  counts  the  number  of  pulses  within  a  given  integration  period  [2],  The  magnitude 
of  the  count  is  proportional  to  the  incident  photon  flux.  The  counter  can  be  controlled  to  count 
up,  down,  or  not  at  all  such  that  a  duobinary  {-1,0,+  1}  modulation  signal  can  be  applied.  In 
addition,  multiple  counters  can  be  connected  to  the  digital  bit  stream  output  of  the  ADC. 

To  decode  the  three  channel  example,  each  of  the  counters  is  set  to  count  up  or  down  during  the 
time  sequences.  For  example,  to  implement  the  first  code  at  ti,  the  first  counter  is  set  to  count 
down,  and  the  second  and  third  counters  to  count  up.  At  the  end  of  the  integration  period,  each 
counter  has  an  estimate  of  the  respective  color  channel.  This  in-pixel  decoding  can  occur  at  MHz 
rates.  At  a  rate  of  1MHz,  200  spectral  channels  can  be  acquired  at  a  rate  of  5kHz  (10,000  times 
greater  than  a  100  frames/sec  imager). 

Mathematically,  the  encoded  light  ( g )  can  be  represented  as  a  product  of  an  encoding  matrix 
(WE)  and  a  feature  vector  (/):  g  =  WEf,  where  /  is  an  Nxl  vector  of  the  spectral  channels,  WE  is 
an  NxN  matrix  with  each  row  corresponding  to  an  orthogonal  code,  and  N  is  the  number  of 
spectral  channels.  In  order  to  recover  the  original  spectral  information,  g  is  multiplied  by  a 
decoding  matrix  ( WD ):  sf  =  WDg  such  that  si  =  WE  WD  where  /  is  the  identity  matrix  and  s  a 
scalar  constant.  For  example,  for  a  vector  of  length  N,  a  Hadamard  matrix  of  rank  N  can  be  used 
for  both  WE  and  WD,  and  s  =  N.  In  practice,  it  is  not  practical  to  use  a  Hadamard  matrix  for  WE 
since  it  is  not  possible  to  apply  a  negative  amplitude  modulation  to  light.  Instead  the  S -matrix  is 
used,  which  contains  only  binary  values  (0,+  l)  and  is  rank  N-l.  To  convert  a  Hadamard  matrix 
to  a  S-matrix:  WE  =  S  =  (1  -  H)/2  [3], 

Technology  Comparison 

Table  1  summarizes  a  technology  comparison  between  using  a  dispersive,  interferometric,  or 
time-encoded  technique  to  acquire  256  spectral  channels  for  a  64  km2  area  at  0.5m  resolution  at  a 
1Hz  update  rate.  Using  a  hyperspectral  system  model,  we  estimate  an  SNR>250  is  needed  for 
90%  detection  at  a  manageable  false  alarm  rate  using  the  adaptive  coherence  estimator  algorithm. 
A  radiance  spectrum  was  calculated  with  MODTRAN  at  5km  altitude,  0.4  to  2.5um  at  lOnm 
resolution.  SNR  calculations  were  calculated  assuming  10cm  optical  aperture,  lOOurad  IFOV, 
50%  overall  optical  system  throughput,  1000x1000  focal  plane  array,  and  detectors  with  85% 
quantum  efficiency  and  1500e-/sec  dark  current. 

Given  that  area  coverage,  SNR,  and  frame  rate  are  interrelated,  the  dispersive  line  scanner  can 
only  meet  the  requirement  for  two  out  of  these  three  figures  of  merit.  In  order  to  scan  an  area  of 
64  km2  at  1Hz  update  rate,  the  per-pixel  dwell  time  is  limited  to  <5us  which  yields  a  SNR  <10. 
In  order  to  achieve  better  SNR,  the  dwell  time  needs  to  be  increased.  To  simultaneously  achieve 
sufficient  SNR  and  area  coverage,  the  update  rate  is  reduced  below  mission  relevance  to  <lmHz. 
In  order  to  meet  SNR  and  update  rate,  the  area  coverage  is  reduced  to  less  than  the  size  of  a 
football  stadium. 

An  interferometer  has  orders  of  magnitude  increase  in  throughput,  however  the  scan  time  is 
limited  by  the  rate  at  which  the  optical  path  differencing  mirror  can  move.  At  a  maximum  rate  of 
2000  steps/s,  the  minimum  dwell  time  to  acquire  the  full  hyperspectral  datacube  is  128ms.  It  is 
not  possible  for  the  interferometer  to  meet  the  requirements  for  update  rate  and  area  coverage 
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because  the  dwell  time  needs  to  be  <5ms.  At  128ms  dwell  time,  the  interferometer  can  meet  the 
area  coverage  and  SNR  at  an  update  rate  of  <0.1  Hz  or  meet  update  rate  and  SNR  with  area 
coverage  of  <5km2. 

In  contrast  to  the  dispersive  and  interferometric  approaches,  the  time-encoded  approach  is 
capable  of  imaging  at  1Hz  update  rate,  25  km2  area  with  SNR=400,  which  is  greater  than  the 
modeled  SNR=250.  The  additional  margin  can  be  used  to  increase  the  update  rate  to  >2Hz  or 
increase  the  area  coverage  to  >100  km2.  In  addition  to  the  high-throughput  fast  scanning 
capability,  the  time-encoded  approach  is  capable  of  flexible  encoding  and  decoding,  which 
enables  multi-mode  operation  that  can  be  programmed  through  software. 


Table  1  Hyperspectral  imaging  technology  comparison. 
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The  time-encoded  multiplexed  approach  enables  flexible  encoding  and  decoding.  At  the  spatial 
light  modulator,  panchromatic  operation  can  be  enabled  by  fixing  the  mirrors,  and  hyperspectral 
resolution  can  be  decreased  to  increase  hypercube  acquisition.  At  the  DFPA,  selected  codes  or 
linear  combinations  of  codes  can  be  decoded.  This  capability  can  be  useful  for  decoding  only 
spectral  bands  of  interest  or  combinations  of  spectral  bands  for  spectral  matched  filtering.  For 
example,  for  256  spectral  bands  approximately  half  are  ignored  due  to  overlap  with  atmospheric 
water  absorption  bands.  The  DFPA  can  selectively  decode  the  good  bands,  whereas  both  the 
dispersive  and  interferometric  methods  need  to  measure  the  entire  spectrum. 

In  FY16  using  internal  funding,  we  built  a  proof-of-concept  prototype  to  demonstrate  the  flexible 
encoding  and  decoding  in  a  laboratory  environment.  The  left  image  in  Figure  2  shows  the 
prototype  test  setup,  which  uses  commercial  of  the  shelf  (COTS)  optical  elements,  a  digital 
micromirror  device  (DMD)  spatial  light  modulator  (SLM)  from  Texas  Instruments,  and  a  custom 
MIT  Lincoln  Laboratory  32x32  8-channel  digital  focal  plane  array.  The  right  image  shows  an 
image  of  a  1300nm  LED  and  a  1450nm  LED,  both  having  a  spectral  width  of  lOOnm.  The  plot 
shows  the  decoded  spectrum  of  two  pixels  of  the  image  with  lOnm  spectral  resolution;  128  codes 
are  used  which  required  acquiring  16  frames,  8  codes  at  a  time.  The  SLM  is  operating  at  10kHz 
modulation  frequency. 
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Figure  2  Table-top  proof-of-concept  prototype  (left)  and  128  channel  spectrum  of  two  pixels  from  an 
image  of  two  LEDs  with  center  wavelengths  of  1300nm  and  1450nm  (right)  and  FWHM  of  lOOnm. 

Figure  3  shows  the  capability  of  flexible  encoding  and  the  tradeoff  between  hypercube 
acquisition  rate  and  spectral  resolution.  In  this  experiment,  each  frame  read  out  from  the  DFPA 
contains  eight  spectral  channels.  Since  the  SLM  is  operating  at  10kHz,  the  total  integration  time 
is  Nx lOOps  where  N  is  the  number  of  spectral  channels  or  codes.  The  hypercube  acquisition  rate 
is  the  frame  rate  divided  by  the  number  of  frames  needed  to  acquire  the  full  hypercube.  For 
example,  to  acquire  128  spectral  channels,  sixteen  frames  are  required  where  eight  spectral 
channels  are  acquired  per  frame.  Decreasing  the  number  of  spectral  channels  decoded  increases 
the  overall  hypercube  rate. 
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Figure  3  Flexible  encoding  enabled  with  a  spatial  light  modulator. 

Figure  4  shows  an  example  of  the  flexible  decoding  that  could  be  enabled  with  a  suitable  DFPA. 
In  this  simulation,  the  DFPA  decodes  the  top  eight  principal  components  and  this  data  is  read  out 
in  a  single  frame.  The  reconstructed  spectrum  shows  good  agreement  with  data  acquired  through 
fully  decoding  the  spectrum  (Figure  3).  By  decoding  the  principal  components,  the  hypercube 
acquisition  rate  can  be  increased  to  the  frame  rate.  For  example,  64  spectral  channels  can  be 
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acquired  at  156  Hz  instead  of  19.5  Hz.  Furthermore,  this  method  can  be  used  to  implement 
spectral  matched  filtering. 


Wavelength  (nm) 

Figure  4  Simulated  decoding  of  principal  components  with  the  digital  focal  plane  array. 

Path  Forward 

The  example  used  for  the  technology  comparison  was  for  wide-area  motion  imaging  from  an 
aerial  platform,  however  the  time-encoded  multiplexed  imaging  technique  is  applicable  to  other 
applications  involving  wide  area  imaging  such  as  hyperspectral  imaging  from  a  spaceborne 
platform.  To  fully  realize  the  potential  of  this  technology,  there  are  still  several  areas  in  need  of 
development.  The  digital  focal  plane  array  (DFPA)  that  was  used  in  the  tabletop  proof-of- 
concept  prototype  was  not  designed  for  multiplexed  imaging,  and  there  are  several  modifications 
that  can  be  made  that  can  improve  functionality.  For  example,  the  control  of  the  in-pixel  counters 
is  currently  global;  per-column  control  would  enable  the  ability  to  do  on-chip  spectral  matched 
filtering.  The  new  architecture  of  this  DFPA  need  to  be  validated  in  a  test  chip  and  eventually 
scaled  up  to  an  appropriate  array  size.  To  enable  wide-area  scanning,  a  fast  steering  mirror  needs 
to  be  synchronized  to  the  spatial  light  modulator  (SLM)  and  DFPA.  Finally,  a  real-world 
demonstration  is  needed  with  data  collected  from  realistic  targets  of  interest. 
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